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INTRODUCTION 

When the Lemm Corporation was contracted to build a multi-million dollar automated 
packaging facility for bulk chemicals, the time frame for the project seemed impossible. Only 
one year was available to design, build, and have operational a brand new plant. This paper 
describes the techniques used to build, from the ground up, the Lemm facility where the 
actual time between the project kickoff meeting and the first production run was only one year 
and 27 days. 

This paper explores the procedures used to procure, test, and install custom automated 
equipment for an ultra fast track project. Important players have specific roles in the project 
and special milestones and management procedures are needed for an ultra fast track 
project. The special techniques used to make the ultra fast track project a reality - such as 
early control system development and in-house testing of all equipment and software are 
described in detail. 

The control system hierarchy from local PLC control to host level inventory and front office 
management are also discussed. Other topics include: Operator training, start-up safety, 
capacity planning, and system diagnostics. 



BACKGROUND 

In 1993, The Lemm Corporation was hired to design, build, and operate a chemical packaging 
facility in Memphis, TN. One of the critical elements of this contract was a short time frame in 
which to design and build the facility. Lemm had one year from the start of the project to have 
the facility operational. The facility was constructed as a "green field" project where no 
building or equipment existed. 



PROJECT MANAGEMENT 

There are three key players on the project management team - the owner, the system 
integrator, and the automation master specialist. The project team should not have a rigid 
hierarchical structure. The owner, the system integrator, and the automation master specialist 
have different responsibilities in each phase of the project. Acceptance of project milestones 
requires the concurrence of all three. 



OWNER 



The owner's role is very important to the ultimate success of the automated system. The 
owner has the business need which must be solved with the automated system. Too often 
the business need gets lost in technical details as the system is being developed. The owner 
has to foot the bill and live with the system after it is installed. If the automated system 
functions perfectly but does not fulfill the business need, it is a failure. 

The owner is in the best position to make decisions which affect the capital cost of the system 
versus ongoing operating expense. For example: Will automating a filling operation at a cost 
of $1 million eliminate enough future operating expenses to make it cost effective? 

The owner's role in this project was assumed by John Lemm of The Lemm Corporation, 
Memphis, TN. 



SYSTEM INTEGRATOR 

The system integrator is a key player in the ultra fast track project. The system integrator 
provides expertise in all aspects of system automation. Various pieces of equipment and 
subsystems will be fused together into a single automated system under the guidance of the 
system integrator. The system integrator must scrutinize the interfaces between vendors both 
mechanical and electrical to make sure that nothing is left out or misinterpreted. One of the 
most common problems with integrating equipment from various suppliers is overlooking 
something at the interface points. Each vendor assumes that the other vendor is handling an 
element and neither of them does. For example, the loading height of a machine is 31" and 
the conveyor feeding it is 30" high, or a handshake signal required by a machine is not 
supplied by the conveyor's PLC. 

The system integrator is also the technical expert who reviews drawings and submittals for 
the owner. The owner seldom has in-depth experience in automated systems and relies on 
the system integrator to interpret the technical issues. The system integrator is responsible 
for the in-house tests performed at the various suppliers and facilitates the flow of information 
between the various contractors. In an ultra fast track project, nothing is assumed. One 
supplier who is less than candid on an issue can seriously affect the project. The system 
integrator is continually monitoring the progress of all suppliers anticipating problems and 
working out solutions before they become obstacles in the project's path. 

The system integrator for the Lemm project was Gerry von Stroh, Tekdata Systems, Parker, 
CO. 



AUTOMATION MASTER SPECIALIST 

This is a new role required by ultra fast track projects. The automation master specialist 
provides in-depth experience in the use of real time modeling for automated systems. 
Modeling is used to replace components of the system needed but not yet available. 

The automation master specialist provides traditional simulation expertise in the early stages 
of the project. The automation master specialist takes the proposed design and equipment 
layout and verifies that it meets the requirements of the project. For example, in the Lemm 
project, the automation master specialist determined the storage requirements by running the 



1992 production year through the model and using the results to determine the size and type 
of equipment needed. 

The automation master specialist adds detail to the simulation model and transforms it into an 
emulation model for use in testing the control software. One of the enabling technologies for 
ultra fast track projects is the ability to start programming the control logic before all of the 
system details are finalized. The process of programming the control logic adds dynamics to 
the system. System elements that were overlooked in the design become obvious on an 
animated display. 

The automation master specialist assists in the in-house tests by using the real time model to 
supply any signals supplied externally to the subsystem under test which are not available. 
The automation master specialist also takes this opportunity to create an "as built" model of 
the subsystem and verify that the subsystem "as built" will properly perform its function in the 
overall system operation. The automation master specialist is responsible for the test of the 
host computer system. The in-house test of the host computer system draws extensively 
upon the scenarios established during the design phase to prove system performance. 

During equipment installation, the automation master specialist sets up the diagnostic monitor 
to assist in testing the proper installation of subsystems by testing the wiring and 
communications links and verifying the operation of the system is the same on the plant floor 
as it was in the in-house tests. The automation master specialist also uses the real time 
model to provide operator training and answer questions that arise during the installation 
process. The real time model is also used as a diagnostic monitor which is left with the 
system to help in ongoing maintainence. 

The automation master specialist for the Lemm project was Max Hitchens, HEI Corporation, 
Carol Steam, IL. 



STRATEGY 

On an ultra fast track project many tasks must occur simultaneously. The building, 
equipment, and software all have to be designed and built on parallel paths. In a traditional 
design/build approach the overall design is completed before the any construction begins. 
The detailed design of a multi-million dollar facility usually takes a year or more. One year 
was all that was available to design, build, and have operational the entire Lemm facility. The 
project team implemented a strategy based upon the following elements: 

1 . Performance contract 

2. Payment based upon demonstrable accomplishments 

3. Rigorous third party testing 

4. Acceptance upon proof of performance 



PERFORMANCE CONTRACT 



An important element of the ultra fast track project is the contract with the suppliers and 
subcontractors. All contracts should be performance based. That is, the contract specifies 
the functioning result of the equipment or subsystem as the deliverable for contract 



completion. For example, filling 2 containers a minute, unloading 50,000 lbs. of bulk product 
per hour, storing 40 unit loads per hour, etc. are specified as the result of contractual 
performance. 

Intermediate payments are made based upon milestones, but the contractual obligation is for 
specific operational results. The contract must have provisions for changes. The ultra fast 
track project requires issuing contracts before the design is complete and all of the detailed 
requirements known. It is necessary to establish a contractual relationship which anticipates 
deviation from the baseline documentation. The deviations should have no cost impact 
unless the changes increase the scope of the project. 

For example, the site preparation may be started before the exact building footprint is 
determined. The contract could include a preliminary drawing with the cost of preparing the 
site to that drawing and the cost ($ per ton) for deviations which require additional work. In 
this example, changes in the building footprint which do not effect the overall amount of dirt 
moved have no effect on the cost of site preparation. 

All changes chargeable to the project require notice and documentation. A subcontractor or 
supplier notifies the owner if a change increasing the scope needs to be made. The owner 
approves the change and the contractor documents the additional work required by the 
change for review at the end of the project. 

IMPORTANT: The contract should require that the additional cost for changes only be paid 
upon completion of contractual obligations. In an ultra fast track project, the project team 
does not have the time to scrutinize change documentation during the intense testing and 
startup periods where nearly all chargeable changes occur. 

One of the toughest elements of the contract is the default clause. In an ultra fast track 
project, a delay by a vendor of only a month can wreak havoc upon the project. A strong 
default clause with liquidated damages should be drafted and the project team must have a 
backup plan at all times to handle a non-performing vendor. 

PAYMENT BASED UPON DEMONSTRABLE ACCOMPLISHMENTS 

The contract is performance based so that fulfillment of the contractual obligation is not 
complete until after the system is installed and operational. Very few suppliers are capable or 
willing to carry the expenditure needed to fulfill the contract for the entire period of time 
necessary to make the system operational. Therefore, progress payments are usually made 
during the course of the project. The progress payments should be structured so that they 
are made based upon demonstrable accomplishments - milestones. 

The progress payments are made upon completion of milestones which provide concrete 
evidence of the results of a value added effort by the supplier. For example, the receipt of the 
steel for the storage system should not be a milestone. However, the fabrication of the steel 
into a storage bay which is then erected and demonstrated is a demonstrable 
accomplishment. 

Properly defining the milestones is very important to the project. The motivating force for a 
corporation is largely economic. A corporation performs work to receive payment. The 
milestone is the goal to which the supplier works toward, and which, when reached provides 
payment as a reward. Pavlovian but very effective. Suppliers or contractors demanding 



prepayments should be avoided. Prepayment removes one of the big levers which the 
project team has to assure the timely performance necessary for an ultra fast track project. 
Experience shows that vendors who are prepaid are more likely to default on the contract by 
not providing timely performance than those who receive payment after the service is 
performed. 

RIGOROUS THIRD PARTY TESTING 

Another necessary element for the ultra fast track project is the ability to monitor the 
performance of all suppliers. One of the biggest problems with ultra fast track projects is lack 
of timely supplier performance. Most contractors are not used to performing to a rapid and 
rigid schedule. 

Tests should be scheduled to demonstrate supplier accomplishments on a regular basis. 
These tests are administered by an independent third party usually the systems integrator or 
automation master specialist. These individuals have experience in the process of producing 
an automated system. The owner knows what the end result should be, but usually does not 
have the experience necessary to determine if the project is on schedule at any given time. 
DO NOT rely on the contractors' statements as to the progress of their work. Continual 
verification of performance by experts must be done throughout the project. 

PROJECT TIP: Some companies, like some individuals, tend to procrastinate on a job. 
Schedule tests in advance of the time when the element is actually required. If the company 
does not have the element ready for test when promised adjust the test schedule accordingly. 
If a contractor routinely slips two weeks on each test, modify the test schedule so that tests 
are run two weeks earlier. An ultra fast track project has insufficient time to modify the 
behavior patterns of a contractor, so that, once the project is started, the team must work with 
the contractor's foibles and characteristics. 

ACCEPTANCE UPON PROOF OF PERFORMANCE 

The completion of any aspect of the project must be based strictly upon proof of its 
performance. Simple delivery or installation is not sufficient. The component or subsystem 
must be delivered, installed, and producing quality at the rate specified. 

The question which must always be answered is: Does the system satisfy the needs of the 
business? Aline may produce at the rate required but not produce quality products. The 
system is not acceptable until it is capable of performing to the requirements of the business. 



PROJECT MILESTONES 



The success or failure of ultra fast track projects hinges upon the definitions and enforcement 
of milestone points. There are six basic milestones for each vendor on the project. 

1. Preliminary drawings. 

2. Control software completion. 

3. In-house test. 

4. Installation completion. 

5. "As built" documentation. 

6. Reliability test. 

These milestones are not always appropriate for every vendor. Vendors which supply smaller 
subsystems will have fewer milestones to which the basic milestones have been reduced. 
Some suppliers of large and complex systems will have additional milestones to assure timely 
and proper performance. 

PRELIMINARY DRAWINGS 

The preliminary drawings are the documents which provide the baseline for what each vendor 
will supply. They are used to review the scope of the vendor's tasks and as discussion focal 
points on the system's operation. 

These drawings are not intended to be final "build to spec" drawings. You cannot expect that 
in 2-3 weeks a complete set of drawings will be produced which detail a complex automated 
system and covers all contingencies. The contract is based upon performance. Acceptance 
of preliminary drawings or agreement with a vendor on an approach did not relieve that 
vendor of the obligation to meet the broad performance criteria of the contract. Acceptance of 
this milestone is the responsibility of the system integrator with concurrence of the owner and 
the automation master specialist. 

CONTROL SOFTWARE COMPLETION 

The completion of this milestone is required before any of the subsystems can begin in-house 
testing. This milestone is one of the key elements to the success of the ultra fast track 
project. The preliminary drawings only present a static picture of the system. The control 
software adds dynamics to that picture. Observing the system in operation on an animated 
computer display enables everyone to see precisely how the system will operate. 

Control software development forces closure on some important issues. The control software 
cannot be completed until all interfaces between vendors have been defined. The control 
software must also operate the system in degraded or manual modes. This assures that the 
operation of the system in automatic, semi-automatic, and manual modes has been tested 
and reviewed. 

Operator interfaces are part of the control software development. Interaction between the 
operator interface and a real time model shows the ergonomics of operating the system. The 



system operation must be completely defined in order to complete development of the control 
software. 

Acceptance of this milestone is the responsibility of the automation master specialist with 
concurrence of the owner and the system integrator. 

NOTE: This milestone and its placement on the time line of the project represents a 
significant departure from the way automated systems have been traditionally developed. 
Completion of the control software early in the project contributed significantly to Lemm's 
success. 

IN-HOUSE TEST 

The in-house test is the test of a subsystem or piece of equipment before its shipment to the 
job site for installation. The test is conducted using the real subsystem and its controls. 

For the in-house test, the equipment is staged at the vendor's location. A close inspection of 
the equipment is made to check for construction quality, e.g. are the welds solid, all pinch 
points guarded, devices and wires labeled properly, etc. The PLC is connected to the 
equipment and the subsystem exercised using the operator interfaces in as close to real 
conditions as possible. The actual equipment operators are used whenever possible. In this 
manner, equipment operators gain experience on operating the equipment and provide 
essential feedback on ergonomic issues. Subsystem functionality and throughput rates are 
verified. 

While the equipment is being exercised it is validated against the model to determine any 
discrepancies between the "as built" and "as designed" subsystem. As each subsystem is 
tested the model is updated to reflect the "as built" subsystem. 

The in-house test is the last milestone before shipping a subsystem for installation. At the 
in-house test, the vendor is required to demonstrate that the subsystem is completely 
functional and ready to ship. 

There are two types of vendors - equipment vendors and software vendors. The equipment 
vendors are required to stage the entire subsystem, connect up the controls, and run the 
actual equipment to demonstrate its functionality. A real time model assists by enabling the 
testing of subsystems independently of each other. The model is used to mimic the operation 
of the remaining subsystems while only the subsystem under test is wired to the actual 
equipment. 

IMPORTANT: As the "as built" model is finished for each subsystem, the model should be 
rerun in simulation mode to verify that none of the changes between the "as designed" 
system and the "as built" system affect the overall performance. 

After a subsystem has passed the in-house test it is ready to be disassembled and shipped to 
site for installation. Acceptance of this milestone is the responsibility of the system integrator 
with concurrence of the owner and the automation master specialist. 



INSTALLATION COMPLETION 



After the in-house test, the equipment is sent to the site for installation as soon as the facility 
is ready to receive it. The installation proceeds with subsystems being installed concurently 
until all equipment and subsystems have been completely installed. The installation is then 
checked for quality and completeness of both mechanical and electrical components. 
Interfaces between subsystems and equipment which have not been connected before 
should be carefully scrutinized. 

The real time model is used to validate field wiring and communications between subsystems. 
As each subsystem is installed on the job site, the wiring is tested for correct re-assembly by 
activating and observing signals on the model running in monitor mode. 

When the entire system is installed and operational, this milestone is complete. Acceptance 
of this milestone is the responsibility of the system integrator with concurrence of the owner 
and the automation master specialist. 



"AS BUILT" DOCUMENTATION 

The "as built" documentation is very important to the long term maintenance and operation of 
an automated facility. Careful review of "as built" documentation requires once again 
inspecting the entire system with the final drawings in hand to verify that the drawings are 
complete and do in-fact match the system as it was installed. The verification and 
acceptance of the "as built" documentation is one of the more time consuming milestones of 
the project. This is the last chance to catch mistakes and errors built into the system. The 
next people who will review the documentation and compare it to the actual system will be 
maintenance personnel and often in a crisis situation were poor documentation can have an 
enormous cost in lost production. 

Acceptance of this milestone is the joint responsibility of the systems integrator, owner, and 
automation master specialist. 



RELIABILITY TEST 

The reliability test is the final milestone of the project. It is the final proof that the system will 
meet the performance requirements of the contract. The reliability test is conducted entirely 
by the system operators. No contractor or equipment suppliers are allowed to intervene. 

The reliability test on the Lemm project consisted of two parts. First, two consecutive shifts 
without a single jam or failure causing operator intervention were required. Any failure or jam 
was noted and the responsible contractor obligated to analyze the failure and correct the 
underlying problem. Just clearing a jam is insufficient to correct the problem. The cause of 
the jam must be determined and the underlying problem fixed. This portion of the test was 
observed by the contractor or supplier. 

The second part of the reliability test required 95% up time at contractual performance levels 
for a five day period. This portion of the test was conducted independently of the contractors 
or suppliers. A log was kept of the activity for the 5 day period, specifically detailing the 
failures that occurred and the actions taken to correct them. This log was reviewed by the 
contractor, owner, and maintenance personnel to determine whether excess downtime was 



caused by equipment flaws which should be fixed by the contractor, insufficient or improper 
spare equipment which is the responsibility of the owner, or improper maintenance techniques 
which can be resolved by additional training. 

Acceptance of this milestone was the responsibility of the owner with concurrence of the 
system integrator and the automation master specialist. 

The time table for the Lemm project is shown as follows: 

LEMM PROJECT TIME TABLE 
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HOST COMPUTER SYSTEM 



The host computer subsystem presents unique challenges. The host computer development 
usually takes as long or longer as the remainder of the project. As such, it is necessary to 
have a plan for starting up the facility in a semi-automatic mode without the host computer. 

The host computer programming requires a team of dedicated professionals. Three 
preliminary documents are required to describe the user interface, architectural layout, and 
automated system interface. These documents should be produced simultaneously and be 
available very early in the project. 



USER INTERFACE 

The user interface document is the description of each user screen with a sample layout. 
Each field on the screen should be described with all possible entries in that field listed and a 
brief description of each. 



ARCHITECTURAL LAYOUT 

A drawing of the system hardware architecture with a description of each piece of equipment 
and its function in the system should be provided. Also, a drawing of system software 
architecture with a description of each piece of software both custom and purchased and its 
function in the system should be provided. 



AUTOMATED SYSTEM INTERFACE 



The automated system interface is the design document which contains all of the interface 
points and messages which are transferred between the host computer and the PLC on the 
factory floor. This document is used by the control logic programmers to interface with the 
host computer. 

The milestones for the host computer subsystem vary greatly depending upon the functions 
required of the host computer. For example, the Lemm host computer handled the following: 

1. Purchasing of consumables. 

2. Inventory tracking 

3. Production scheduling 

4. Customer orders 

5. Bar code printers 

6. Bar code scanners 

7. RF terminals 

8. Operator terminals 

9. General purpose printers (reports, etc.) 

10. Speciality printers (Bills of Lading, etc.) 

The owner, system integrator, and automation master specialist must review the system 
requirements with the host computer vendor and establish intermediate milestones for host 
computer subsystem development. Milestones should be no further than 2 months apart. 

In cases where real time data or control is exchanged between the PLC and host computer, a 
PLC and a real time model needs to be set up for use by the host computer programmers. 

HOST COMPUTER IN-HOUSE TEST 

The final milestone for the host computer system prior to its shipment and installation in the 
field is the in-house test. This test is crucial to the success of the entire project because it is 
the last (and generally the only) opportunity for the entire computer and control hierarchy to 
be tested. A lab must be set up with all of the host computer equipment including operator 
terminals, printers, barcode printers, RF terminals, etc. Also connected to the host computer 
is the PLC (or network of PLCs), man-machine interfaces, and real time models to duplicate 
the equipment being installed in the field. 

The host computer in-house test has a dual purpose. It checks out the functionality of the 
host software and more importantly, it verifies that the automated system as a whole will meet 
the business requirements which prompted the automated system in the first place. For 
example, can orders be entered on the host computer, scheduled for production, 
automatically produced, and shipped to the customer smoothly with the tools provided at the 
host computer level and the plant level? 



Actual production personnel should be used in the host computer in-house test if possible. 
The test provides an opportunity for production personnel to be trained in advance and gain 
experience in operating the system. It also demonstrates that the host computer system can 
be operated by ordinary production personnel. The operators should be instructed not to 
"baby" the system. They should be requested to try and "break" the system with the only 
stipulation being that they carefully keep track of the actions they take so that those actions 
can be duplicated by the host computer programmers in order to fix any problem uncovered. 

In the case of the Lemm project, the computer in-house test was three days of actual 
production. This approach was very successful. After the first two days of testing it was 
evident that the host computer software was not robust enough to withstand the Lemm 
operators. So the first attempt at the in-house test was abandoned and the host computer 
programmers left with a 100 item punch list to fix. The second attempt a month later was 
successful. The hardware configuration used by Lemm for the host computer in-house test 
follows: 



HOST COMPUTER IN-HOUSE TEST CONFIGURATION 




The completion of the host computer in-house test proves that the host computer software, 
PLC logic, and man machine interfaces can withstand use by actual operators under field like 
conditions. 



ULTRA FAST TRACK PROJECT TECHNIQUES 

The ultra fast track project incorporates some new techniques which are not normally used on 
projects. These techniques represent major deviations from standard project procedures. 

AUTOMATION MASTER SPECIALIST 

An automation master specialist was added to the project team. In most projects, the owner 
works with a system integrator to manage the project. This pairing has one flaw, system 
integrators seldom have detailed experience in real time modeling. An automation master 
specialist has experience with testing automated systems at all levels from the use of 
simulation to validate the design, the use of emulation to test control software, and the use of 
on-line diagnostics to verify "as builts" and proper system operation. 



EARLY CONTROL SOFTWARE DEVELOPMENT 



The milestone for control software completion was moved up on the time line from after 
installation where it is normally placed to before in-house testing. This forces important 
decisions on how the system is going to operate to be made while the equipment is being 
designed and fabricated instead of after the equipment is installed and an attempt is being 
made to make it operational. 

In traditional projects, the software programming is not started until the equipment has been 
built because the equipment is used to debug the software. This approach slows down the 
project in two ways. First, the software debugging time is placed on the critical path of the 
project. Since the time required to debug the control system is typically 30% of the control 
software development time, placing this time on the critical path of the project can extend the 
overall completion time by up to 30%. 

The second way the project is slowed down by using the equipment to debug the software is 
that it ties up the equipment so that it cannot be used for other purposes such as production 
tuning and testing. By not having the equipment available for production tuning and testing 
until after the software has been debugged adds another 20% or more to the project 
completion time. 

RIGOROUS IN-HOUSE TESTING 

All subsystems are rigorously tested at the contractor's facility before they are allowed to be 
shipped to the job site. At the contractor's facility all of the equipment and personnel which 
the contractor has available are available to work on the subsystem. After the subsystem is 
shipped to site, only a small field team with limited tools accompany it. Consequently, the 
solution to problems encountered in the field are hampered by the lack of available tools and 
personnel and the necessity of working around other vendors. 

DO NOT succumb to inevitable complaints by vendors that they are wasting time by 
assembling the subsystem for the in-house test, then disassembling it, only to reassemble it 
again on the job site. The biggest mistake made on the Lemm project was, after giving 
vendors a punch list of items to be fixed as a result of the in-house test, obliging their request 
to be allowed to fix these items while the equipment was being disassembled for shipment. 
The equipment arrived at the job site with a majority of the punch list items uncorrected. 

DO repeat the in-house test as often as necessary until the subsystem performs 100% before 
it leaves the vendors facility. There will be enough problems to fix in the field when the 
subsystems are connected together for the first time without the necessity of dealing with 
problems which could have been more easily fixed in the vendor's facility. 

REAL TIME MODELING 

Real time modeling is a critical tool necessary for the simultaneous execution of different 
tasks on the project time line. The real time modeling software used on the Lemm project 
was Automation Master which is the only modeling software package available which can be 
used for the entire life cycle of the project. An Automation Master model operates in various 
modes at different stages of the project. 



SIMULATION MODE 



In simulation mode, the real time model acts as a stand alone system. The model is used to 
test the proposed system design and provide capacity planning for the automated facility. 

For example, on the Lemm project, one of the unknown elements was the storage subsystem 
which consisted of an AS/RS crane feeding flow rack storage. The storage requirement was 
a 10 day supply of product to cover any process perturbations and change over situations. 
There were four different sizes of storage bays required by the product. The model was used 
in simulation mode to run one years worth of historical production to determine the size and 
mix of storage bays required. 

Real time simulation is not traditionally used for design type simulations. The advantage of 
using a real time simulator during the design phase of the project is the reusability of the 
model. Since any detailed model takes time to develop, maximizing its use is important. 
Using traditional next event simulators, the simulation model can be used in the design 
process to make basic design decisions but it cannot be used as a tool to test software or 
train operators. Real time simulation is not the fastest or easiest way to do a "quick and dirty" 
design phase simulation, but due to its intrinsic reusability, it is the most economical approach 
for projects which are carried through to completion. Ultra fast track» projects have a funded 
budget do not depend upon a specific design to get the project approval. 

The set up of Automation Master in simulation mode is shown below: 













Automation 
Master 













In simulation mode, Automation Master acts as a stand alone simulator and does not require 
any other equipment to operate. 

EMULATION MODE 

One of the unique characteristics of a real time simulator like Automation Master is its ability 
to accept decisions made by external devices. This allows the model to emulate the 
automated equipment to the control system. The control system is programmed as it 
normally would be and the model used to test it. Thus, the control software programming can 
be started early in the project since the real time model replaces the actual equipment for 
debugging purposes. 

In the Lemm project, the emulation model was built section by section as soon as the design 
for the section was stable. The vendors of the software were supplied the model for use in 
testing of their software. The completion and test of the control software under emulation was 
a significant project milestone with a substantial payment to reward its completion. 

Emulation models are extremely detailed, the stroke of each air cylinder is input into the 
model with the time required to extend and retract. All discrete components - photo eyes, limit 
switches, etc. are placed in the model exactly where they will be positioned on the actual 
equipment. 



The process of modeling the equipment and programming the software brings out many 
detailed design questions which can be answered and problems solved before the equipment 
has been completely fabricated. 

All of the operator stations and push button panels are included in the model. By laying out 
the operator panels in the model, the ergonomics and operating sequences are brought to the 
attention of the programmers. The panels can be easily rearranged before any holes have 
been punched in cabinets. 

The man machine interfaces can also be tested using the emulator. The control software 
runs in the PLC, the Automation Master model acts as the equipment, and the man-machine 
interface is used to interact with the simulated equipment. This configuration is also used for 
operator training. 

An example of the set up of Automation Master in emulation mode is shown below: 



Automation 
Master 



In emulation mode, Automation Master is connected to the PLC which controls the equipment 
and mimics the operation of the equipment by reading the output image from the PLC and 
responding with the input image containing the state of the inputs as if the actual equipment 
were attached. 

MONITOR MODE 

Automation Master acts as a diagnostic monitor on the installed system. In this mode, the 
real time model runs in parallel with the actual system. Each activity in the real system is 
compared with the activity in the model. If the activities concur or are within the specified 
tolerances, the system is operating normally. If the activities are different or outside of 
specified tolerances, an alarm is generated. Since the model is "perfect," there are no jams 
or equipment failures in the model. Any jam or equipment failure causes an activity which 
does not correspond to the activity in the model and generates an error on the diagnostic 
monitor. 



An example of the set up of Automation Master in monitor mode is shown below: 
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In monitor mode, Automation Master is connected to the PLC that controls the equipment. 
The state of all outputs and inputs are read and compared with the activity in the real time 
model. Errors are displayed on the graphics display. 

The monitor mode also has a field verification feature which enables the "as built" system to 
be compared with the model. The verification prints out the difference between each activity 
in the system and the activity in the model. This allows the model to be modified to 
correspond exactly to the real system and close the loop in the modeling process. 

The verification feature causes each discrete activity - photo eye, limit switch, etc. to print the 
difference between the actual activity of the equipment and the activity in the model to a log. 
The log is then used to create an "as built" model to match the "as built" system. The log 
provides the difference in both time, and if applicable, distance between the signal transition 
in the model and the signal transition in real system. 

The variance log will give a LEAD time value, if the model signal transition occurs before the 
real signal transition, or a LAG time value if the model signal transition occurs after the real 
signal transition. An example of the verification log is shown below: 



10/11 13:02:39 ON PE01101 
10/11 13:02:44 OF PE01102 
I . I 



-01.177 LAG 
+00.422 LEAD 



DATE & TIME 
POLARITY — 
SIGNAL NAME 

TIME VARIANCE 

DIAGNOSTIC MESSAGE 

SUPPLEMENTAL INFORMATION 



- 1.345U 
+ 0.844U 



The date and time field contains the date and time from the system clock when the log data 
was output. The polarity field contains the polarity of the transition either ON or OFF which 
caused the log. The signal name is the mnemonic for the signal used in the model. The 
time variance is the time in seconds that the model signal differed from the real signal. A 



LEAD diagnostic message indicates the signal in the model occurred before the real signal. A 
LAG diagnostic message indicates the signal in the model occurred after the real signal. The 
supplemental information field contains the variance between the model signal and the real 
signal expressed as a distance. The distance variance is the distance which must be added 
to the model to exactly match the "as built" system. 

The real time model is refined and improved throughout the simulation and emulation 
processes. After verification the model is identical to the real system, so all tests performed 
using the model have identical results to the real system. 

MIXED MODE OPERATION 

Another of the unique advantages in using a real time simulator is the ability to run a model 
partially in simulation mode and partially in emulation mode. For example, the filling systems 
were custom built for the Lemm project. Their proper operation was critical to the overall 
system performance. During the in-house test of the filling system, the PLC-5 which ran the 
material handling system and communicated with the VAX system was connected to the 
SLC-500 which ran the filler. The model was attached to the PLC-5 and the actual equipment 
attached to the SLC-500. In this manner the actual operation of the filler could be tested as 
well as its interaction with the rest of the system. The filler in-house test configuration follows: 

FILLER IN-HOUSE TEST CONFIGURATION 
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SYSTEM ARCHITECTURE CONSIDERATIONS 

The system configuration must allow for parallel development and independent testing on 
ultra fast track projects. This section describes the system architecture used on the Lemm 
project to independently develop and test the various subsystems. 

The system architecture was structured to enable the in-house testing at multiple vendor sites 
to be conducted concurrently. Each subsystem was built independently with its own remote 
I/O rack and control cabinet. This enabled the subsystem to be tested by simply plugging in 



the PLC to the remote I/O connector on the rack. An Automation Master model was used to 
supply any interface signals the subsystem needed. 

During the in-house test, the model and PLC were connected to the subsystem to be tested. 
All other equipment, except for the subsystem under test was emulated. The I/O from the 
subsystem under test was real with the Automation Master model supplying the I/O for the 
remaining equipment. In this manner, each subsystem could be tested independently. At the 
same time the impact that the subsystem would have on the remainder of the system could 
be determined. 

The ladder logic for each subsystems was written by different programmers and run on one 
PLC. In order to enable simultaneous development, each programmer used different 
program and data files for their logic. For example: the host computer interface was written in 
program file #1 and used internal files - B3, T4, C5, N7, etc. The bulk handling logic was 
placed in program file #2 and used internal files - B13, T14 C15, N17, etc. The conveyor 
logic was written in program file #3 and used internal files - B23, T24 C25, N27, etc. All 
interfaces between the different programs were documented and used data from common 
internal files. 

A spare of each PLC was purchased with a rack and power supply so that the different 
software developers could have access to a PLC to test their logic and sufficient PLCs were 
available to conduct in-house tests and logic tests simultaneously. During the intensive 
in-house test times another PLC was borrowed so that sufficient PLCs were available to 
operate the subsystems and perform software tests simultaneously. 

Atypical test configuration for a subsystem in the in-house tests is shown below: 

SUBSYSTEM IN-HOUSE TEST CONFIGURATION 
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The PC with the Automation Master model and a PLC with the program was taken to the 
vendor's site where it was connected to the subsystem for the in-house test. 



